feat(evals): add deterministic VQA dataset generation by ruthwikdasyam · Pull Request #3488 · dimensionalOS/dimos

ruthwikdasyam · 2026-08-17T07:58:36Z

Contribution path

Closes DIM-1418

Problem

DimOS needs reproducible visual-question datasets generated from recorded camera frames.

Solution

Add dimos evals vqa generate and dimos evals vqa run workflows for deterministic multiple-choice VQA datasets. Questions are constrained by family,
answers come from private Moondream evidence, and
outputs include lossless PNG assets plus audit metadata.

Added 3 deterministic families - presence, horizontal_detection, object_count -> which are pre-built using primitive methods (moondream here) - so can just call the method to get the solution - making it deterministic.

How to Test

dimos evals vqa generate go2_short.db --image-index 100

dimos evals vqa run ~/.local/state/dimos/datasets/vqa/go2_short-frames --model gpt-4o-mini

uv run --no-sync pytest dimos/evals/vqa

AI assistance

OpenCode with GPT-5.6 Sol assisted with implementation, tests, documentation, and review.

Checklist

I have read and approved the CLA.

codecov · 2026-08-17T08:05:16Z

Codecov Report

❌ Patch coverage is 91.09312% with 66 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/evals/vqa/generate.py	79.88%	29 Missing and 5 partials ⚠️
dimos/evals/vqa/suite.py	80.48%	9 Missing and 7 partials ⚠️
dimos/evals/vqa/families.py	91.54%	3 Missing and 3 partials ⚠️
dimos/evals/vqa/test_contracts.py	98.45%	4 Missing ⚠️
dimos/evals/runner.py	84.61%	2 Missing ⚠️
dimos/evals/vqa/author.py	91.30%	1 Missing and 1 partial ⚠️
dimos/cli/vqa.py	96.00%	0 Missing and 1 partial ⚠️
dimos/evals/test_evals.py	96.66%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #3488      +/-   ##
==========================================
+ Coverage   77.19%   77.28%   +0.09%     
==========================================
  Files        1260     1267       +7     
  Lines      119881   120620     +739     
  Branches    10581    10632      +51     
==========================================
+ Hits        92539    93222     +683     
- Misses      24276    24318      +42     
- Partials     3066     3080      +14

Flag	Coverage Δ
OS-ubuntu-24.04-arm	`72.19% <91.09%> (+0.11%)`	⬆️
OS-ubuntu-latest	`74.08% <91.09%> (+0.10%)`	⬆️
Py-3.10	`74.08% <91.09%> (+0.10%)`	⬆️
Py-3.11	`74.08% <91.09%> (+0.10%)`	⬆️
Py-3.12	`74.08% <91.09%> (+0.10%)`	⬆️
Py-3.13	`74.08% <91.09%> (+0.10%)`	⬆️
Py-3.14	`74.08% <91.09%> (+0.10%)`	⬆️
Py-3.14t	`74.08% <91.09%> (+0.10%)`	⬆️
SelfHosted-Large	`30.04% <31.44%> (+0.01%)`	⬆️
SelfHosted-Linux	`35.16% <31.44%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/cli/dimos.py	`100.00% <100.00%> (ø)`
dimos/cli/test_vqa.py	`100.00% <100.00%> (ø)`
dimos/evals/types.py	`91.95% <100.00%> (+0.18%)`	⬆️
dimos/models/vl/base.py	`49.55% <100.00%> (ø)`
dimos/cli/vqa.py	`96.00% <96.00%> (ø)`
dimos/evals/test_evals.py	`97.67% <96.66%> (-0.17%)`	⬇️
dimos/evals/runner.py	`74.70% <84.61%> (+0.42%)`	⬆️
dimos/evals/vqa/author.py	`91.30% <91.30%> (ø)`
dimos/evals/vqa/test_contracts.py	`98.45% <98.45%> (ø)`
dimos/evals/vqa/families.py	`91.54% <91.54%> (ø)`
... and 2 more

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mintlify · 2026-08-17T17:45:07Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
dimensional	🟢 Ready	View Preview	Aug 17, 2026, 5:45 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

greptile-apps · 2026-08-18T18:36:19Z

Greptile Summary

Adds commands for generating and running standalone visual-question-answering datasets. The implementation produces portable image assets, public multiple-choice cases, private labels, and audit records, and evaluates model answers through the shared evaluation runner.

Confidence Score: 5/5

Safe to merge based on the final review findings.

No blocking failure remains.

T-Rex Logs

What T-Rex did

Identified the authored reproduction script empty-output-repro.py as the basis for the proof-of-work.
Reran the reproduction and captured the command output in empty-output-02-after.log.
Documented the dependency blockers encountered during setup, including missing numpy and a pyaudio compile error due to portaudio.h.
Observed that earlier attempts showed isolated runtime imports missing project dependencies.
Noted that no tracked source was modified; only untracked proof artifacts were created.

_{Ran code and verified through T-Rex}

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

TomCC7 · 2026-08-19T03:41:54Z

+
+    model_config = ConfigDict(extra="forbid", frozen=True)
+
+    family: Literal["presence", "horizontal_direction", "object_count"]


you might want to check this: https://pydantic.dev/docs/validation/latest/concepts/unions/#discriminated-unions

currently every family has same payload. when distinct fields are introduced, ill change them to discriminated unions

TomCC7 · 2026-08-19T03:51:34Z

+            f"{self.inputs}\nChoices: {json.dumps(self.choices)}\nAnswer with exactly one choice."
+        )
+        outputs = rig.ask(context, prompt)
+        answer = _parse_choice(outputs, self.choices)


maybe we should always ask llm to provide structured output (like a json) and validate against them uniformly to avoid these custom cleanup/parsing logic

yeah, thats a good use of structured output. wil change

TomCC7 · 2026-08-19T03:58:21Z

+
+    def evaluate(self, rig: EvalRig) -> EvalResult:
+        image = Image.from_file(self.image_path)
+        context = [] if rig.blind else cast("list[dict[str, Any]]", image.agent_encode())


what's the point of doing a vqa with the blind option? basically no information provided?

good question. I added this like a benchmark sanity check. you know how dataset have questions like is object x visible? and answer is always yes, coz we cannot auto-generate true negative questions.
It should score around 30% or less in blind mode, to make sure dataset is good. If more, then its bad. a good sanity check

TomCC7 · 2026-08-19T04:11:05Z

+    case_by_id = _unique_by_id(cases, "case")
+    label_by_id = _unique_by_id(labels, "label")
+    if case_by_id.keys() != label_by_id.keys():
+        missing_labels = sorted(case_by_id.keys() - label_by_id.keys())
+        missing_cases = sorted(label_by_id.keys() - case_by_id.keys())
+        raise ValueError(
+            f"VQA case/label IDs do not match: missing_labels={missing_labels}, "
+            f"missing_cases={missing_cases}"
+        )


this whole checking seems to imply that there's a 1-1 correspondance between case and label. in that case why not just use 1 jsonl instead of 2?

cases.jsonl is public - image path, question and choices, labels.jsonl is private with case id ad expected answer. so theyhave different visbility, but one-to-one correspondence is intentional

spomichter · 2026-08-19T16:52:28Z

+
+import typer
+
+app = typer.Typer(help="Generate and evaluate standalone visual question-answering datasets.")


all cli change should be in dimos/cli so we can track everything

Got it, moved to dimos/cli/vqa.py

spomichter · 2026-08-19T16:54:09Z

+def _write_frame(output: Path, frame: _GeneratedFrame) -> None:
+    (output / "assets").mkdir(parents=True, exist_ok=True)
+    frame_audit = output / "audit" / f"frame-{frame.index:06d}"
+    frame_audit.mkdir(parents=True, exist_ok=True)
+


No hard coded path generation / making directories. this will work for a git cloned dimos but not for a library installed dimos. Needs to be done properly and save VQA sets to .local/ the same place mem2 does.

dataset root already uses STATE_DIR. the asset and audit directories here are relative parts of dataset schema.

…ration-1 # Conflicts: # uv.lock

…ration-1 # Conflicts: # dimos/cli/dimos.py

ruthwikdasyam added 4 commits August 16, 2026 22:49

feat(evals): add single-frame VQA generation

b0ca0da

feat(evals): expand deterministic VQA generation

591778a

feat(evals): generate multi-frame VQA datasets

f7445cf

feat(evals): add object-count VQA family

104ce86

ruthwikdasyam added 2 commits August 17, 2026 09:46

fix(evals): restore default VQA output path

11b4e45

docs(evals): consolidate VQA guide

0c86ebb

mintlify Bot deployed to staging - docs August 17, 2026 17:45 View deployment

fix(evals): harden VQA dataset generation

c6f09f4

mintlify Bot deployed to staging - docs August 18, 2026 08:14 View deployment

fix(evals): align VQA image encoding

9079838

mintlify Bot deployed to staging - docs August 18, 2026 18:17 View deployment

ruthwikdasyam marked this pull request as ready for review August 18, 2026 18:26

ruthwikdasyam requested review from Dreamsorcerer, leshy, mustafab0, paul-nechifor and spomichter as code owners August 18, 2026 18:26

ruthwikdasyam mentioned this pull request Aug 18, 2026

feat(evals): support point-cloud VQA families #3492

Open

1 task

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 19, 2026

TomCC7 reviewed Aug 19, 2026

View reviewed changes

spomichter reviewed Aug 19, 2026

View reviewed changes

Comment thread dimos/evals/vqa/primitives/moondream.py Outdated

spomichter reviewed Aug 19, 2026

View reviewed changes

spomichter requested changes Aug 19, 2026

View reviewed changes

paul-nechifor reviewed Aug 20, 2026

View reviewed changes

Comment thread docs/usage/vqa.md Outdated

paul-nechifor reviewed Aug 20, 2026

View reviewed changes

Comment thread docs/usage/vqa.md Outdated

paul-nechifor reviewed Aug 20, 2026

View reviewed changes

Comment thread docs/usage/vqa.md Outdated

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 20, 2026

mintlify Bot deployed to staging - docs August 20, 2026 07:50 View deployment

ruthwikdasyam added 2 commits August 20, 2026 00:51

fix(evals): improve VQA dataset handling

673ac4f

Merge remote-tracking branch 'origin/main' into ruthwik/feat/vqa-gene…

ce9b032

…ration-1 # Conflicts: # uv.lock

ruthwikdasyam force-pushed the ruthwik/feat/vqa-generation-1 branch from 488fb0b to ce9b032 Compare August 20, 2026 07:53

mintlify Bot deployed to staging - docs August 20, 2026 07:54 View deployment

refactor(cli): centralize VQA commands

3b41a03

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 20, 2026

ruthwikdasyam added 2 commits August 21, 2026 19:36

feat(evals): use structured VQA responses

b8acb10

Merge remote-tracking branch 'origin/main' into ruthwik/feat/vqa-gene…

27ec44c

…ration-1 # Conflicts: # dimos/cli/dimos.py

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 22, 2026

mintlify Bot deployed to staging - docs August 22, 2026 02:58 View deployment

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 22, 2026


		model_config = ConfigDict(extra="forbid", frozen=True)

		family: Literal["presence", "horizontal_direction", "object_count"]


		import typer

		app = typer.Typer(help="Generate and evaluate standalone visual question-answering datasets.")

Uh oh!

Conversation

ruthwikdasyam commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution path

Problem

Solution

How to Test

AI assistance

Checklist

Uh oh!

codecov Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mintlify Bot commented Aug 17, 2026

Uh oh!

greptile-apps Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

T-Rex Logs

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ruthwikdasyam commented Aug 17, 2026 •

edited

Loading

codecov Bot commented Aug 17, 2026 •

edited

Loading

greptile-apps Bot commented Aug 18, 2026 •

edited

Loading